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The concept of entropy rate for a dynamical process on a graph is introduced. We study diffusion processes 
where the node degrees are used as a local information by the random walkers. We describe analitically and 
numerically how the degree heterogeneity and correlations affect the diffusion entropy rate. In addition, the 
entropy rate is used to characterize complex networks from the real world. Our results point out how to design 
optimal diffusion processes that maximize the entropy for a given network structure, providing a new theoretical 
tool with applications to social, technological and communication networks. 
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Entropy is a key concept in statistical thermodynamics HI, 
in the theory of dynamical systems Q], and in information 
theory 0]. In the realm of complex networks J3,|5t], the en- 
tropy has been used as a measure to characterize properties of 
the topology, such as the degree distribution of a graph |fj, or 
the shortest paths between couples of nodes (with the main in- 
terest in quantifying the information associated with locating 
specific addresses J3l, or to send signals in the network Jit]. 
Alternatively, various authors have studied the entropy asso- 
ciated with ensembles of graphs, and provided, via the appli- 
cation of the maximum entropy principle, the best prediction 
of network properties subject to the constraints imposed by a 
given set of observations 13, Hi 11]. An approach of this type 
plays the same role in networks as is played by the Boltzmann 
distribution in statistical thermodynamics (]]]. 

The main theoretical and empirical interest in the study of 
complex networks is in understanding the relations between 
structure and function. Besides, many of the interaction dy- 
namics that takes place in social, biological and technological 
systems can be analyzed in terms of diffusion processes on top 
of complex networks, e.g. data search and routing, informa- 
tion and disease spreading 0, @]. It is therefore of outmost 
importance to relate the properties of a diffusion process with 
the structure of the underlying network. 

In this Letter, we show how to associate an entropy rate 
to a diffusion process on a graph. In particular, we consider 
processes such as biased random walks on the graph that can 
be represented as ergodic Markov chains. In this context, the 
entropy rate is a quantity more similar to the Kolmogorov- 
ASinai entropy rate of a dynamical system M12lll3ll . than to the 
entropy of a statistical ensemble HIQ]. Differently from the 
network entropies previously defined, the entropy rate of dif- 
fusion processes depends both on the dynamical process (the 
kind of bias in the random walker) and on the graph topol- 
ogy. We provide the analytical expression that describes the 
entropy rate in scale-free networks as a function of the bias in 
the walk, and of the degree distribution and correlations. We 
show how the values of the entropy rate can provide useful 
information to characterize diffusion processes in real-world 
networks. In particular, a maximum value of entropy is found 



for different types of the bias in the diffusion processes, de- 
pending on the network structure. 

Let us consider a connected undirected graph with N nodes 
(labelled as 1, 2, N) and K links, described by the adja- 
cency matrix A = {a^ } We limit our discussion to diffu- 
sion processes on the graph that can be represented as Markov 
chains In particular, we consider the case of biased ran- 
dom walks in which, at each time step, the walker at node i 
chooses one of the first neighbors of i, let say j, with a prob- 
ability proportional to the power a (a g R) of the degree kj. 
Such biased random walk corresponds to a time-invariant (the 
rule does not change in time) Markov chain with a transition 
probability matrix II, with elements: 



J2j a ijkf 



(1) 



Notice that II depends on either the graph topology and the 
kind of stochastic process we are considering. The exponent 
a allows to tune the dependence of the diffusion process on 
the nodes' degree. When a^Owe are introducing in the ran- 
dom movement of the particle a bias towards high- {a > 0) 
or low-degree (when a < 0) neighbors. On the other hand, 
when a = the standard (unbiased) random walk is recov- 
ered. Since the walker must move from a node to somewhere, 
we have J^j Tji = 1> mus II is a stochastic matrix. If Wi(t) 
is the probability that the random walker is at node i at time t 
(with J^iLi w i(t) = 1 Vi), then the probability Wj(t + 1) of 
its being at j one step later is: Wj(t+1) = Yli 7T ji w i(t)- Writ- 
ing the probabilities Wi (t) as a A^-dimensional column vector 
w(<) = ( w\(t), W2(t) . . . Wjv(i) ) T > the rule of the walk can 
be expressed in matricial form as: w(t + 1) = Ilw(t). In 
the case of an undirected and connected network, the Perron- 
Frobenius theorem Il4tl assures that the dynamics described 
by Eq. (Q]i is an ergodic Markov chain ^ . This means that the 
Markov chain has a unique stationary distribution w*, such 
that lirrit^oo n*w(0) = w* for any initial distribution w(0). 
In other words, any initial distribution of the random walker 
over the nodes of the graph will converge, under the dynamics 
of Eq. ([T), to the same distribution w*. 

The dynamical properties of the above diffusion processes 



2 



over the graph can be accounted by evaluating the entropy rate 
of the associated Markov chain that, in the case of an ergodic 
Markov chain, is given by y|]: 



h = - ^ TTjj ■ W* ln(TTji) 



(2) 



h3 



The value of h measures how the entropy of the biased ran- 
dom walk grows with the number of hops. This means that 
we can practically represent the typical sequences of length 
?i generated by the diffusion process by using approximately 
n ■ h information units. In different words, h measures the 
spreading of a set of independent random walkers, in terms of 
number of visited nodes. 

To evaluate h for a given graph we need to calculate the 
stationary probability distribution w*. For this purpose, we 
consider the probability Wi_>j (t) of going from node i to node 
j in t time steps, 



E 



'h3l 
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(3) 



Since the network is undirected we have a,j = 
Hence, the relation between the two probabilities Wi^.j(t) 
and Wj-,i(t) can be written as: 



c i k?W i ^ j (t) = c j k"W j ^ i (t) , 



(4) 



where c, — £ ^ a ij kf ■ The above relation implies that for the 
stationary distribution w* the equation Cikfw* 



holds, and hence w* reads: 



Cjkfw* 



(5) 



By plugging expressions (fl~|i and (0 into the definition of en- 
tropy (0, we finally get a closed form for the entropy rate of 
degree-biased random walks on the graph: 



fcf Ej <Hjkf Hkf) - k? Cl Info 

Si**? 



(6) 



We notice that h depends on the the kind of bias in the ran- 
dom walker and also on the graph topology. In the following, 
we first evaluate analytically the entropy rate of unbiased and 
biased random walks on scale-free (SF) graphs with a power- 
law degree distribution P k ~ fc -7 , and 7 > 2 J3,|5t]. Then, 
we study the entropy in networks from the real world. 

Unbiased Random Walks. - In the particular case a = 0, the 
transition probability reads TTji = o^/fci, and the stationary 
distribution is easily obtained as: w* — jj?. Substituting this 
expression in Eq. (0 and changing the sum over node indexes 
into a sum over degree classes, we can write the entropy rate of 
a unbiased random walk on a network with degree distribution 
Pk as: 



N 
2K 



(fcln(fc)) 



(7) 
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FIG. 1 : Entropy rate h of unbiased random walks on SF networks 
with N — 10 5 nodes as a function of the exponent 7 of the degree 
distribution. Numerical results (circles) are compared with the two 
analytical curves corresponding to Eq. {8]l (dashed line) and to the 
limit iV — > 00 (solid line). 



In the case of SF networks of size N, the value of h can 
be easily expressed as a function of 7 and N taking into ac- 
count that the maximum degree of the network is k max ~ 
koN 1 ^ 1 ^ 1 ^, with ko being the minimum degree of a node. 
From Eq. (0, and approximating k as a continuum variable, 
we get: 



/ 1 ( 7 ,A r ) = ln(fc ) 



7-2 ( 7 _l)(JV= 



1) 



(8) 



The above expression diverges for SF networks when 7 — > 2. 
Conversely, when 7 > 2 SF networks have a finite entropy in 
the thermodynamic limit: h(j) = ln(fcrj) + ^^rg- 

In order to check the analytical results we have constructed 
ensembles of 10 2 SF networks with N = 10 5 nodes and dif- 
ferent values of 7. We have obtained numerically the station- 
ary distribution w*, and computed the entropy directly from 
Eq. (0. The results, averaged over the ensemble of networks, 
are reported in Fig. Q] as a function of 7. We notice a good 
agreement between numerics and Eq. (0. 

Biased Random Walks.- Let us now concentrate on degree- 
biased diffusion (a ^ 0). In this case, the entropy rate of 
Eq. (0 can be re-written by changing again the sums over 
node indexes into sums over degree classes, as: 



£ fe k a P k (C k ln(C fc ) - ak' a P k , M ln(fc') 



Z k C k k°P k 



where C k — k £\/ k P k > k , and P f 



k 'k 



(9) 

is the conditional 



probability that a link from a node of degree k ends in a node 
with degree k . We notice that the entropy rate of biased ran- 
dom walks depends on the degree distribution of the network, 
P k , and on the conditional probabilities P k i k . In the partic- 
ular case of a network with no degree-degree correlations we 
can write P k > k = kP k /(k), and the expression for the entropy 
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reduces to: 



h=(l-a) 



{k a+1 ln(fc)) 
(k a+1 ) 



In 



a+l\ 



(k) 



(10) 



This expression only depends on the degree distribution of 
the network. For SF networks, we get in the the continuum- 
degree approximation: 



h(Tf,a,N) 
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When N — > 00, the entropy rate in SF networks with 7 < 2 + 
a diverges as h ~ ln(iV). On the other hand, when 7 > 2 + a, 
the entropy rate in the limit N — > 00 is finite and equal to: 



h(7,a) 



1 — Q 



hi 



M7 



7 — a 



(12) 



Such an expression, valid in infinite size limit, shows a mono- 
tone growth of the entropy h(j, a) with the degree-bias a, 
with h tending to infinity as a — > (7 — 2) - . More interest- 
ingly, the entropy rate in finite networks, Eq. ill U . shows a 
single maximum at a value of a that depends on 7. This result 
is a consequence of the interplay between diffusion process 
and network topology. It indicates that, for a given network, 
is possible to maximize the entropy of the process by oppor- 
tunely tuning the bias a of the walker. 

To check the above analytical expressions we have com- 
puted numerically the entropy rate of degree-biased random 
walkers on computer-generated uncorrected SF networks, as 
we did for the unbiased case. In Fig.|2]a we report the entropy 
rate as a function of the degree bias a for SF networks of size 
N = 10 5 . In Fig.|2]b we show the scaling of h with the system 
size N, in SF network with 7 = 3. In both cases Eq. (fTTT i is 
in good agreement with the numerical results reproducing the 
qualitative behavior of ft, as a function of a (being the global 
maximum of h well reproduced) and N (being both the diver- 
gence of h, for a > 7 — 2, and the asymptotic finite value of 
h, when a < 7 — 2, correctly reproduced) II 1511 . 

Real Networks.- Up to now, we focused on the entropy 
rate of biased random walks on SF networks. However, 
real networks are not perfect scale-free and, more impor- 
tantly, show additional important structural properties such 
as degree-degree correlations, motifs and community struc- 
tures |0, [H. Now we propose to characterize a real network 
by studying different diffusion processes on top of it, and 
finding the optimal value of the bias that maximizes the en- 
tropy. As reference system, we compare the entropy rate h 
of a-biased random walks on the network, with the entropy 
rate h Rand obtained, from Eq. ([Toi l, for a randomized ver- 
sion of the network, with the same degree sequence of the 
real one fl6Tl . For this purpose, we have analyzed 10 different 
networks reported in Table [I] corresponding to thre different 
functional classes where diffusion of data, rumors, viruses and 
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FIG. 2: (color online), (a) Entropy rate h, as a function of a, for 
a-biased random walks on SF networks with iV = 10 nodes and 
7 = 2.6, 3, 3.4. Symbols represent the values of h found numer- 
ically, while the lines are the corresponding analytical predictions 
h(-y, a, N) of Eq. Jilt , (b) Entropy rate h for a-biased random 
walks on SF networks with 7 — 3, as a function of the system size 
N and for several values of a. Again, symbols are the results of 
numerical simulations, while the lines correspond to Eq. i ll It . 



diseases, takes place, namely ( i) transportation, ( ii) technolog- 
ical/communication and (Hi) social networks. 

In Fig. [3]we report, for six of the networks, the results ob- 
tained as a function of the bias parameter a. Two different 
behaviors emerge clearly for a > 12611 . namely the entropy 
of the real network h is either larger or smaller than h rand for 
all the range of positive values of a. In table |T] we summarize 
this result by reporting the ratio h/h Rand for a = 1 (linear 
bias). We found that social networks have always h > h Rand , 
while the other networks have h < h Rand , with the excep- 
tion of Internet routers. This difference in the entropy rate has 
its roots mainly on the different types of degree-degree cor- 
relations of the network, and points out that assortativity fa- 
cilitates the spread of the diffusion. The optimal degree-bias, 
a opt that produces the maximal entropy rate is also reported in 
TableQ] The results indicate that for assortative networks (e.g. 
social networks) the maximal entropy rate is obtained with a 
super-linear diffusion, while for dissasortative networks a opt 
is located in the sub-linear bias region. 
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TABLE I: Properties of the 10 real networks analyzed. N is the 
number of nodes in the giant connected component, (fc) is the av- 



erage degree. The ratio of entropy rates, h/h 



is reported for 



a linear (a = 1) degree-biased random walk. Finally we report the 
optimal value of a that maximizes h for each network. 
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FIG. 3: Entropy rate h for a-biased random walks on six of the net- 
works in Table |I] (filled circles). Such entropy rate is compared with 
that obtained for the randomized version of the network (full circles). 
Both entropy rates are shown as a function of the degree-bias param- 
eter a. 



Summing up, in this Letter, we have introduced the entropy 
rate of degree-biased random walks on networks, a measure 
that is particularly suited to capture the interplay between net- 
work structure and diffusion dynamics. We have studied the 
dependence of the entropy rate with the topology of synthetic 
and real networks, in particular with the heterogeneity of de- 
gree distributions and the nature of the degree-degree correla- 
tions. The results indicate how it is possible to tune the bias 
in the random walk in order to maximize the entropy rate on 
a given topology. The method introduced can find useful ap- 
plications to cases where diffusion in complex networks is the 
mechanism at work, such as in the search of efficient algo- 



rithms for data search in the WWW, in the improvement of 
information dissemination in social networks, or in the de- 
sign of large impact virus/antivirus spreading in computer net- 
works. The approach adopted here can be easily generalized 
to other types of diffusion processes and to more general net- 
work topologies, such as weighted graphs and also, with some 
appropriate modifications to directed and unconnected graphs. 

The authors are indebted to M. Chavez and H. Touchette 
for many helpful discussions on the subject. 
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